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Preface  &  Acknowledgements 


Welcome  to  our  Ninth  Annual  Acquisition  Research  Symposium!  This  event  is  the 
highlight  of  the  year  for  the  Acquisition  Research  Program  (ARP)  here  at  the  Naval 
Postgraduate  School  (NPS)  because  it  showcases  the  findings  of  recently  completed 
research  projects — and  that  research  activity  has  been  prolific!  Since  the  ARP’s  founding  in 
2003,  over  800  original  research  reports  have  been  added  to  the  acquisition  body  of 
knowledge.  We  continue  to  add  to  that  library,  located  online  at 

www.acquisitionresearch.net,  at  a  rate  of  roughly  140  reports  per  year.  This  activity  has 
engaged  researchers  at  over  60  universities  and  other  institutions,  greatly  enhancing  the 
diversity  of  thought  brought  to  bear  on  the  business  activities  of  the  DoD. 

We  generate  this  level  of  activity  in  three  ways.  First,  we  solicit  research  topics  from 
academia  and  other  institutions  through  an  annual  Broad  Agency  Announcement, 
sponsored  by  the  USD(AT&L).  Second,  we  issue  an  annual  internal  call  for  proposals  to 
seek  NPS  faculty  research  supporting  the  interests  of  our  program  sponsors.  Finally,  we 
serve  as  a  “broker”  to  market  specific  research  topics  identified  by  our  sponsors  to  NPS 
graduate  students.  This  three-pronged  approach  provides  for  a  rich  and  broad  diversity  of 
scholarly  rigor  mixed  with  a  good  blend  of  practitioner  experience  in  the  field  of  acquisition. 
We  are  grateful  to  those  of  you  who  have  contributed  to  our  research  program  in  the  past 
and  hope  this  symposium  will  spark  even  more  participation. 

We  encourage  you  to  be  active  participants  at  the  symposium.  Indeed,  active 
participation  has  been  the  hallmark  of  previous  symposia.  We  purposely  limit  attendance  to 
350  people  to  encourage  just  that.  In  addition,  this  forum  is  unique  in  its  effort  to  bring 
scholars  and  practitioners  together  around  acquisition  research  that  is  both  relevant  in 
application  and  rigorous  in  method.  Seldom  will  you  get  the  opportunity  to  interact  with  so 
many  top  DoD  acquisition  officials  and  acquisition  researchers.  We  encourage  dialogue  both 
in  the  formal  panel  sessions  and  in  the  many  opportunities  we  make  available  at  meals, 
breaks,  and  the  day-ending  socials.  Many  of  our  researchers  use  these  occasions  to 
establish  new  teaming  arrangements  for  future  research  work.  In  the  words  of  one  senior 
government  official,  “I  would  not  miss  this  symposium  for  the  world  as  it  is  the  best  forum 
I’ve  found  for  catching  up  on  acquisition  issues  and  learning  from  the  great  presenters.” 

We  expect  affordability  to  be  a  major  focus  at  this  year’s  event.  It  is  a  central  tenet  of 
the  DoD’s  Better  Buying  Power  initiatives,  and  budget  projections  indicate  it  will  continue  to 
be  important  as  the  nation  works  its  way  out  of  the  recession.  This  suggests  that  research 
with  a  focus  on  affordability  will  be  of  great  interest  to  the  DoD  leadership  in  the  year  to 
come.  Whether  you’re  a  practitioner  or  scholar,  we  invite  you  to  participate  in  that  research. 

We  gratefully  acknowledge  the  ongoing  support  and  leadership  of  our  sponsors, 
whose  foresight  and  vision  have  assured  the  continuing  success  of  the  ARP: 

•  Office  of  the  Under  Secretary  of  Defense  (Acquisition,  Technology,  &  Logistics) 

•  Director,  Acquisition  Career  Management,  ASN  (RD&A) 

•  Program  Executive  Officer,  SHIPS 

•  Commander,  Naval  Sea  Systems  Command 

•  Program  Executive  Officer,  Integrated  Warfare  Systems 

•  Army  Contracting  Command,  U.S.  Army  Materiel  Command 

•  Office  of  the  Assistant  Secretary  of  the  Air  Force  (Acquisition) 
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•  Office  of  the  Assistant  Secretary  of  the  Army  (Acquisition,  Logistics,  & 
Technology) 

•  Deputy  Director,  Acquisition  Career  Management,  U.S.  Army 

•  Office  of  Procurement  and  Assistance  Management  Headquarters,  Department 
of  Energy 

•  Director,  Defense  Security  Cooperation  Agency 

•  Deputy  Assistant  Secretary  of  the  Navy,  Research,  Development,  Test  & 
Evaluation 

•  Program  Executive  Officer,  Tactical  Aircraft 

•  Director,  Office  of  Small  Business  Programs,  Department  of  the  Navy 

•  Director,  Office  of  Acquisition  Resources  and  Analysis  (ARA) 

•  Deputy  Assistant  Secretary  of  the  Navy,  Acquisition  &  Procurement 

•  Director  of  Open  Architecture,  DASN  (RDT&E) 

•  Program  Executive  Officer,  Littoral  Combat  Ships 

We  also  thank  the  Naval  Postgraduate  School  Foundation  and  acknowledge  its 
generous  contributions  in  support  of  this  symposium. 

James  B.  Greene  Jr.  Keith  F.  Snider,  PhD 

Rear  Admiral,  U.S.  Navy  (Ret.)  Associate  Professor 
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Chair:  Lorna  B.  Estep,  Deputy  Director  of  Logistics,  Air  Force  Materiel  Command 

Discussant:  Katherine  Schinasi,  Independent  Consultant,  former  Managing 

Director,  U.S.  Government  Accountability  Office 

Basis  for  a  Rational  Defense:  Acquiring  the  Right  Capability 

J.  David  Patterson,  National  Defense  Business  Institute,  The  University  of 
Tennessee 

Analysis  of  Alternatives:  Keys  to  Success 

John  F.  Schank,  RAND  Corporation 

Lorna  B.  Estep — Ms. Estep  is  the  deputy  director  of  logistics  at  the  Directorate  of  Logistics  and 
Sustainment,  Headquarters  Air  Force  Materiel  Command,  Wright-Patterson  Air  Force  Base,  OH.  Ms. 
Estep  is  a  member  of  the  Senior  Executive  Service.  She  is  responsible  for  the  Materiel  Support 
Division  of  the  Supply  Management  Activity  Group,  a  stock  fund  with  annual  sales  of  $7  billion.  She 
directs  a  wide  range  of  logistics  services  in  support  of  Air  Force  managed  spare  parts,  to  include 
transformation  programs,  requirements  determination,  budgeting,  acquisition,  provisioning, 
cataloging,  distribution  and  data  management  policy.  She  also  provides  supply  chain  management 
policy,  guidance  and  direction  in  support  of  headquarters,  air  logistics  centers,  and  U.S.  Air  Force 
worldwide  customers. 

Estep  started  her  career  as  a  Navy  logistics  management  intern.  She  has  directed  the  Joint 
Center  for  Flexible  Computer  Integrated  Manufacturing,  was  the  first  program  manager  for  Rapid 
Acquisition  of  Manufactured  Parts,  and  has  served  as  technical  director  of  Information  Technology 
Initiatives  at  the  Naval  Supply  Systems  Command.  In  these  positions,  she  has  developed  logistics 
programs  for  the  Department  of  Defense,  implemented  one  of  the  first  integrated  and  agile  data- 
driven  manufacturing  systems,  and  directed  the  development  of  complex  technical  data  systems  for 
the  Navy. 

Katherine  Schinasi — Ms.  Schinasi  is  currently  an  independent  consultant.  Until  its  sunset  in  201 1, 
Ms.  Schinasi  was  a  commissioner  with  the  Commission  on  Wartime  Contracting  in  Iraq  and 
Afghanistan,  a  position  she  was  appointed  to  by  the  Senate  Majority  Leader.  The  bipartisan 
Commission  was  established  and  charged  with  recommending  contract-related  improvements  in 
contingency  environments.  Ms.  Schinasi  continues  to  provide  advice  and  assistance  to  Congress  on 
implementing  the  Commission’s  recommendations. 

Beginning  in  2009,  Ms.  Schinasi  was  a  senior  advisor  to  The  Conference  Board,  a  non-profit 
research  firm  in  New  York.  As  such  she  was  responsible  for  conducting  and  commissioning 
academic-level  research  on  21st  century  information  and  communications  networks  and  their  impact 
on  society  and  culture.  She  co-edited  The  Linked  World:  How  ICT  is  Transforming  Societies, 
Cultures,  and  Economies  and  continues  to  provide  assistance  to  the  business  community  on  those 
issues. 

From  1978  until  March  2009,  she  was  employed  by  the  Government  Accountability  Office, 
serving  the  last  six  years  as  managing  director  for  acquisition  and  sourcing  management.  Her 
portfolio  included  operations  in  the  Departments  of  Defense,  State,  and  Homeland  Security,  and 
cross-governmental  acquisitions.  In  her  position,  Ms.  Schinasi  testified  frequently  before 
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congressional  committees  on  topics  of  defense  trade  and  investment,  export  controls,  the  acquisition 
of  services  and  major  developmental  systems,  and  the  relationship  between  government  and  private 
sector  businesses. 

She  was  a  frequent  lecturer  at  the  Defense  Acquisition  University,  the  Industrial  College  of  the 
Armed  Forces,  and  the  Naval  Postgraduate  School.  Her  team  of  180  people,  based  in  five  U.S. 
locations,  was  responsible  for  recommendations  that  led  to  billions  of  dollars  in  savings  and  improved 
government  operations,  including  those  involving  government-contractor  relationships.  She  was 
appointed  to  the  federal  Senior  Executive  Service  in  1998. 

Ms.  Schinasi  received  an  MA  in  international  relations  from  the  School  of  International  Service, 
American  University,  and  a  BA  in  government  and  politics  from  the  University  of  Maryland. 
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Analysis  of  Alternatives:  Keys  to  Success 


John  F.  Schank — Schank  (MS,  Operations  Research,  University  of  Pennsylvania)  joined  RAND  in 
1 972.  He  has  been  involved  in  a  wide  range  of  analysis  that  has  included  industrial  base  analyses, 
cost  analyses,  and  studies  of  manpower,  personnel,  training,  and  logistics  issues  at  both  the  Service 
and  joint  level.  He  has  led  or  co-led  numerous  projects  for  the  U.S.  Navy,  the  U.S.  Coast  Guard,  the 
Office  of  the  Secretary  of  Defense,  and  the  United  Kingdom’s  Ministry  of  Defence.  Among  his  larger, 
more  influential  studies  were  a  project  on  sustaining  the  U.S.  capability  to  design  nuclear  submarines, 
which  resulted  in  the  addition  of  more  than  $9  million  to  the  Navy’s  submarine  design  budget  to  begin 
work  on  the  next  class  early;  an  analysis  of  alternatives  (AoA)  on  transit  of  special  forces  through 
littoral  areas;  and  an  AoA  on  the  next-generation  sea-based  strategic  deterrent.  His  other  recent 
studies  have  dealt  with  increasing  aircraft  carrier  forward  presence,  sustaining  key  skills  in  the  UK 
naval  industry,  workload-  and  workforce-management  practices  in  U.S.  Navy  shipyards,  and  relating 
littoral  combat  ship  performance  to  mission  package  inventories,  home  ports,  and  installation  sites. 

He  has  authored  or  coauthored  close  to  100  documents  for  RAND. 

Abstract 

An  analysis  of  alternatives  (AoA)  is  an  important  step  in  the  acquisition  process.  It  is  one  of 
the  first  places  where  the  requirements  community  and  the  resource  community  must  come 
together  to  identify  a  preferred  material  solution  that  can  fill  a  shortfall  in  desired  operational 
capability.  AoAs  inform  Service,  Office  of  the  Secretary  of  Defense  (OSD),  and  congressional 
decision-makers  on  the  relative  cost  effectiveness  of  viable  alternatives  that  meet  the  desired 
capabilities. 

Based  on  conducting  several  AoAs  for  the  various  military  Services,  RAND  has  formed  a  set 
of  important  considerations  when  conducting  an  AoA: 

•  develop  a  thorough  study  plan  that  considers  a  range  of  alternatives  and  baselines, 

•  form  effect  relationships  with  oversight  committees; 

•  conduct  trade-off  analyses  and  examine  sensitivities; 

•  have  a  flexible  analysis  methodology; 

•  display  results  that  are  easily  and  quickly  understood;  and 

•  recognize  and  estimate  technical,  design,  and  production  risks. 

Introduction 

An  analysis  of  alternatives  (AoA)  is  an  important  step  in  the  acquisition  process.  It  is 
one  of  the  first  places  where  the  requirements  community  and  the  resource  community  must 
come  together  to  identify  a  preferred  materiel  solution  that  can  fill  a  shortfall  in  desired 
operational  capability.  AoAs  inform  Service,  Office  of  the  Secretary  of  Defense  (OSD),  and 
congressional  decision-makers  on  the  relative  cost  effectiveness  of  viable  alternatives  that 
meet  the  desired  capabilities. 

AoAs  use  an  objective,  systematic,  and  unbiased  process  to  identify,  evaluate,  and 
document  the  costs  and  mission  effectiveness  of  alternative  systems.  AoAs  help  establish 
critical  mission  characteristics  and  system  performance  requirements,  and  then  identify 
potential  alternative  systems  that  can  satisfy  those  requirements.  An  AoA  develops  an 
analytic  framework  and  methodology  to  measure  the  operational  performance  and  total 
costs  of  each  alternative.  It  then  develops  appropriate  cost-effectiveness  measures  and 
presents  the  findings  and  recommendations  to  the  acquisition  executives. 

The  objective  of  this  paper  is  not  to  describe  and  define  how  to  conduct  an  AoA. 
There  are  numerous  sources  for  that  information  (e.g.,  Defense  Acquisition  University,  n.d.; 
Office  of  Aerospace  Studies,  Air  Force  Materiel  Command,  2008).  Nor  is  the  intent  to 
present  some  new  revelations  on  the  conduct  of  AoAs.  Rather,  the  objective  is  to  coalesce  a 
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number  of  important  aspects  of  AoA  execution  that  are  self-evident  to  us  all,  but  that  often 
fall  through  the  cracks  during  the  complex  and  often  time-constrained  conduct  of  the 
analysis. 

The  paper  describes  several  important  principles  that  can  help  contribute  to  the 
success  of  an  AoA.  They  are  based  on  insights  drawn  from  several  AoAs  conducted  by 
RAND  (Moore,  2004;  Kennedy  et  al. ,  2006),  including  those  examining  the  Joint  High  Speed 
Vessel  (JHSV;  Schank  et  al.,  2006),  Sea-Based  Strategic  Deterrent  (SBSD;  Schank  et  al. 
2010),  the  Coast  Guard’s  Offshore  Patrol  Cutter  (OPC;  Arena,  2011),  and  various  assets 
under  consideration  by  the  Special  Operations  Command  (USSOCOM;  DeLuca  et  al.,  2008, 
2009).  Success  is  measured  not  just  by  the  acceptance  of  the  AoA  by  the  decision 
authorities,  but  by  how  thoroughly  it  identifies  alternatives;  evaluates  the  costs, 
effectiveness,  and  risks  of  those  options;  and  presents  the  findings  in  a  way  that  decision¬ 
makers  can  understand  the  trade-offs  between  operational  requirements,  system  capability, 
and  cost. 

The  paper  is  timely  since  the  Government  Accountability  Office  (GAO)  recently 
criticized  the  conduct  of  previous  AoAs  (GAO,  2009).  Based  on  reviewing  over  20 
completed  AoAs,  the  GAO  found  that  many  AoAs  did  not  consider  a  robust  range  of 
alternatives  or  adequately  consider  the  cost,  schedule,  and  performance  trade-offs.  Many 
AoAs  did  not  fully  understand  the  operational  requirements  and  lacked  sufficient  knowledge 
of  available  and  future  technologies.  Because  knowledge  of  technologies,  design  risks,  and 
production  impacts  was  lacking,  many  AoAs  based  their  performance,  cost,  and  schedule 
estimates  on  optimistic  assumptions.  Also,  the  GAO  criticized  the  OSD  and  the  Services  for 
not  providing  adequate  guidance  on  the  conduct  of  AoAs. 

Three  Fundamental  Questions  All  AoAs  Address 

An  AoA  basically  answers  three  questions: 

•  What  are  the  desired  operational  capabilities  (Do  What?) 

•  What  systems  and  technologies  are  or  will  be  available  to  provide  the  desired 
capabilities  (With  What?) 

•  What  are  the  costs  and  performance  of  the  alternatives  (How  Well)? 

These  three  questions  can  be  answered  through  six  high-level  tasks  as  shown  in 
Figure  1 .  The  first  task  establishes  how  the  systems  will  operate  and  scenarios  for 
measuring  operational  performance.  The  second  task  identifies  key  performance 
parameters  (KPPs)  and  the  measures  of  effectiveness  (MOEs)  and  performance  (MOPs) 
that  will  help  in  the  evaluation  of  the  alternatives.  In  parallel  to  the  first  two  tasks,  Task  3 
identifies  viable  alternatives  that  typically  combine  various  platforms  and  systems,  such  as 
propulsion  or  sensors.  Tasks  4  and  5  then  use  the  scenarios,  MOEs,  and  MOPs  to  estimate 
the  effectiveness  and  total  costs,  respectively,  of  each  alternative.  Task  6  combines  the  cost 
and  effectiveness  analyses  and  presents  the  findings  and  recommendations  to  the  Service 
and  OSD  decision  authorities.  Each  basic  task  is  decomposed  into  several  subtasks. 
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Figure  1.  Six  Tasks  Help  Address  Three  Basic  Questions 

Most  AoAs  are  conducted  in  a  teaming  environment  with  a  lead  study  director  and 
study  organizations  supported  by  subject-matter  experts  in  several  key  areas.  For  example, 
in  the  AoAs  RAND  has  conducted,  cost  and  engineering  organizations  from  the  Naval  Sea 
System  Command,  other  federally  funded  research  and  development  centers  (FFRDCs), 
and  even  private-sector  organizations  have  been  part  of  the  AoA  team. 

The  six  basic  tasks  in  Figure  1  seem  straightforward  enough,  but  there  are 
underlying  complexities  that  must  be  addressed  for  an  AoA  to  be  successful.  These  include 
the  following: 

•  develop  a  thorough  study  plan  that  considers  a  range  of  alternatives  and 
baselines; 

•  form  effect  relationships  with  oversight  committees; 

•  conduct  trade-off  analyses  and  examine  sensitivities; 

•  have  a  flexible  analysis  methodology; 

•  display  results  that  are  easily  and  quickly  understood;  and 

•  recognize  and  estimate  technical,  design,  and  production  risks. 

In  the  following  sections,  we  discuss  lessons  that  we  have  learned  in  discussing 
each  of  these  complexities. 

Develop  a  Thorough  Study  Plan 

Figure  2  shows  that  an  approved  study  plan  is  required  before  an  AoA  can  start.  An 
approved  Initial  Capabilities  Document  (ICD)  defines  the  gap  in  operational  capability  and 
the  requirement  for  a  solution  to  fill  that  gap.  The  ICD  may  also  suggest  general  alternatives 
that  can  provide  the  materiel  solution  based  on  functional  studies  conducted  during  the 
development  of  the  ICD. 
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Figure  2.  AoAs  and  the  Defense  Acquisition  Process1 

(DAU,  n.d.) 


The  Director,  Cost  Assessment  and  Program  Evaluation  (CAPE)  provides  study 
guidance  for  the  AoA.  This  guidance,  along  with  the  ICD,  feeds  into  the  development  of  the 
study  plan. 


The  study  plan  provides  guidance  and  objectives  for  the  conduct  of  the  AoA.  It  is  the 
agreement  between  the  AoA  study  director  and  the  Service  and  OSD  oversight 
organizations  on  what  will  be  accomplished  and  how  it  will  be  done.  The  study  plan 
identifies 


•  the  scope  of  the  analysis,  the  ground  rules,  and  assumptions; 

•  the  concepts  of  operation  and  the  scenarios  for  evaluating  mission  effectiveness; 

•  an  initial  set  of  MOEs  and  MOPs  for  evaluating  mission  performance; 

•  the  broad  classes  of  alternatives  and  the  system  trade-offs  within  alternatives; 

•  the  range  of  cost  elements  for  estimating  the  total  cost  of  each  alternative; 

•  the  roles  and  responsibilities  of  the  organizations  that  make  up  the  study  team; 

•  the  responsibilities  of  the  oversight  groups;  and 

•  the  schedule  and  deliverables  during  the  conduct  of  the  AoA. 

The  study  plan  must  be  thorough,  clear,  and  concise.  A  good  study  plan  is  a 
prerequisite  for  a  successful  AoA.  Adequate  time  and  resources  must  be  devoted  to  develop 
the  study  plan.  Guidance  and  inputs  from  the  subject-matter  experts,  as  well  as  from  the 
Service  and  OSD  oversight  organizations,  are  essential  for  the  development  of  a  study  plan 
that  will  serve  as  the  map — agreed  to  by  all  stakeholders — guiding  the  analysis. 

Form  Effective  Relationships  with  Oversight  Committees 

Effective  relationships  with  Service  and  OSD  oversight  groups  are  important  for  the 
AoA  study  director  and  study  team.  Ultimately,  the  oversight  groups  must  approve  the  AoA. 


1  This  is  Figure  1  from  the  Defense  Acquisition  University  website.  The  figure  and  a  nice  overview  of  the 
development  and  organization  of  an  AoA  study  plan  is  at  https://acc.dau.mil/ComminityBrowser.aspx?id=314769. 
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These  groups  will  also  provide  guidance  during  the  AoA  and  may  require  the  evaluation  of 
additional  alternatives  or  trade-offs  between  system  performance  and  operational 
requirements. 

Critical  to  the  interactions  between  the  AoA  study  team  and  the  oversight  groups  is 
the  establishment  of  a  steering  group.  The  steering  group  should  provide  guidance  as 
issues  arise  as  well  as  access  to  subject-matter  experts  and  various  stakeholders.  The 
steering  group  should  work  with  the  study  director  to  approve  decisions  as  analysis 
progresses. 

The  steering  group  should  be  comprised  of  representatives  from  research  and 
development  centers  who  can  inform  the  study  team  of  the  status  of  various  technologies. 
There  should  also  be  representatives  from  the  program  office,  costs  organizations, 
operational  commands,  and  support  and  logistics  groups.  The  group  should  have  a 
chairperson  appointed  by  the  Service  or  OSD.  The  study  director  should  meet  with  the 
steering  committee  on  a  monthly  basis  to  discuss  issues  and  potential  solutions  to  those 
issues. 


Again,  effective  interactions  and  relationships  between  the  study  director  and  the 
oversight  groups  are  essential  for  a  successful  AoA. 

Conduct  Trade-Off  Analyses 

The  primary  purpose  of  an  AoA  is  to  evaluate  the  cost-effectiveness  of  the 
alternatives.  However,  the  AoA  should  go  beyond  evaluating  basic  alternatives.  It  must 
examine  trade-offs  between  various  systems  or  subsystems  and  between  operational 
requirements  and  the  ultimate  costs  of  meeting  those  requirements.  For  example,  an  AoA 
should  address  the  following  questions: 

•  How  do  operational  requirements  drive  system  performance? 

•  What  is  the  operational  impact  of  relaxing  various  system  requirements? 

•  What  new  technologies  (and  associated  risks)  are  needed  to  achieve  operational 
goals? 

•  How  does  cost  relate  to  desired  mission  performance? 

•  Which  attributes  drive  costs  and  risks? 

Figures  3  and  4  show  a  trade-off  analysis  that  was  conducted  during  the  JHSV  AoA. 
The  alternatives  evaluated  during  that  AoA  were  basically  different  types  of  hulls,  such  as 
mono-hulls,  multi-hulled  designs,  and  surface  effect  ships.  But  the  choice  of  the  propulsion 
system  also  impacted  cost  and  performance.  Figure  3  shows  how  the  choice  of  a  propulsion 
system  resulted  in  different  force  levels  that  would  be  required  to  deliver  a  combat  force  into 
a  theater  in  a  given  number  of  days  (a  duration  known  as  closure  time).  As  shown  in  the 
figure,  diesel  propulsion  systems  needed  more  vessels  for  a  given  closure  time  compared  to 
gas  turbine  (GT)  or  combined  diesel  and  gas  turbine  systems  (CODAG). 
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Figure  3.  Propulsion  System  Trade-Off  Between  Closure  Time  and  Number  of 

Vessels 

However,  as  shown  in  Figure  4,  further  analysis  showed  that  the  total  acquisition 
costs  for  the  force  were  largely  the  same  across  the  three  types  of  propulsion  systems  for  a 
given  closure  time. 


Figure  4.  Propulsion  System  Trade-Off  Between  Closure  Time  and  Acquisition  Cost 

Figure  5  shows  another  trade-off  analysis  that  was  informative  during  the  SBSD 
AoA.  That  AoA  had  to  address  not  only  the  type  of  platform  but  also  the  size  and  number  of 
missile  tubes.  The  analysis  depicted  in  Figure  5  identified  breakpoints  for  missile  tube 
diameter.  For  example,  for  Tomahawk  missiles  there  was  no  difference  in  the  number  of 
missiles  that  could  fit  into  tubes  between  94  and  103  inches  in  diameter.  Since  larger  tube 
diameters  meant  bigger  hull  forms,  this  trade-off  analysis  helped  define  the  cost-effective 
size  of  the  platform  under  various  missile  payload  assumptions. 
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Future  Missile  Tube  Diameter  (in.) 


Figure  5.  Number  of  Missiles  in  Different  Diameter  Tubes 

Have  a  Flexible  Analysis  Methodology 

Things  will  change  during  an  AoA.  New  alternatives  will  be  defined.  Additional  MOEs 
and  MOPs  will  be  identified.  New  scenarios  or  mission  requirements  will  be  added  or 
modified.  New  questions  will  arise.  The  original  study  plan  will  morph  as  the  AoA 
progresses.  Because  of  these  inevitable  changes,  the  AoA  must  develop  a  flexible  and 
transparent  methodology  for  evaluating  effectiveness  and  estimating  costs  that  can  adapt  as 
changes  arise.  This  methodology  must  be  understood  and  approved  by  the  steering  and 
oversight  groups.  Typically,  several  simple,  easily  understood  models  are  better  than  a 
single  large,  complex  model  that  clouds  the  relationships  between  variables. 

Display  Results  in  a  Way  That  is  Easily  Understandable 

AoAs  typically  deal  with  complex  issues.  Multiple  factors  and  variables  surrounding 
operational  requirements,  system  performance,  and  cost  are  often  interrelated.  The  AoA 
must  capture  this  complexity  yet  clearly  show  the  impact  of  different  decisions  and  choices. 
The  charts,  graphs,  and  tables  that  display  results  of  the  analyses  are  the  mechanisms  the 
AoA  uses  to  transmit  the  findings  and  recommendations.  These  displays  must  be  easy  to 
understand  and  decipher. 

Figure  6  shows  a  display  that  we  used  in  one  AoA  to  show  the  effectiveness  of 
various  alternatives  in  several  mission  areas.  Although  no  single  alternative  proved  most 
effect  in  all  mission  areas,  the  operational  value  of  alternatives  A  and  B  clearly  stood  out 
when  considering  the  range  of  missions. 


ACQUISITION  RESEARCH  PROGRAM: 
CREATING  SYNERGY  FOR  INFORMED  CHANGE 


-  339- 


Counter  Drug  Migrant  Coastal 

Interdiction  Security 


Fisheries 


100  ■ 

■  O  A  ioo- 

■  <$►  A,  B  ioo  I 

■  O  A  ioo  ■ 

■  O  A  ioo  t 

■  ♦  B 

90  ■ 

90  ■ 

90  ■ 

90  ■ 

■  ♦  B  90  ' 

80  ■ 

■  O  D 

■  O  D 

■  O  D 

„  _  70  '■ 

80  ■ 

■  ♦  B 

80  ■ 

80  ■ 

80  ■ 

■  O  C 

60  ■' 

■  <0  E,  F 

70  ■ 

50  ■' 

70  ■ 

■  O  E,F  70- 

■  O  C  70- 

40  ■ 

■  o  c 

60  ■ 

■  O  F 

30  ■ 

■OF 

60  ■ 

60  ■ 

60  ■ 

1  Op 

..  20  " 

•=  50  ■ 

■  <0  C&D 

♦  E  10 

50  ■ 

50  ■ 

L  50  ■ 

40  ■ 

0  ■■ 

Presence 


■  ♦  B 


8  £ 


O  F 

0>  &D 


Figure  6.  Comparing  the  Effectiveness  of  Platforms  Across  Missions 

Table  1  shows  another  display  that  related  various  MOEs  and  MOPs  across  the 
various  alternatives.  Here,  alternatives  A,  E,  and  G  all  performed  equally  well.  Finally,  Figure 
7  shows  a  display  used  to  relate  life-cycle  cost  to  mission  effectiveness  for  various 
alternatives.  In  this  example,  alternatives  K  and  L  were  shown  to  be  very  costly  for  the 
capability  they  provided. 

The  study  director  must  be  creative  in  putting  together  the  displays  that  show  the 
results  of  the  AoA.  Typically,  it  is  better  to  have  several  simple  charts  that  build  on  each 
other  rather  than  a  single,  complex  chart  where  relationships  are  not  obvious. 


Table  1.  Comparing  Effectiveness  of  Various  Alternatives 


Alternative 

Measures  of  Effectiveness 

Measures  of  Performance 
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Figure  7.  Comparing  Life  Cycle  Cost  to  Mission  Effectiveness 
Recognize  and  Estimate  Risks 

One  of  the  GAO’s  criticisms  of  previous  AoAs  was  their  lack  of  appreciation  for,  and 
consideration  of,  the  various  risks  involved  with  different  alternatives  or  system  solutions. 
Ignoring  or  underestimating  risks  can  lead  to  incorrect  decisions  that  result  in  future  cost  and 
schedule  growth.  The  AoA  must  clearly  understand  and  convey  the  state  of  various 
technologies,  which  often  requires  that  the  studies  be  informed  by  subject-matter  experts  in 
various  fields.  While  typically  associated  with  technologies,  risks  can  also  arise  from 
inadequacies  in  the  industrial  base,  especially  in  the  second-tier  vendor  base.  Risks  may 
also  arise  from  other  programs  the  AoA  system  is  dependent  upon.  For  example,  although 
the  Future  Combat  System  (FCS)  for  the  Army  made  heroic  assumptions  on  the  readiness 
of  key  technologies  needed  for  program  success,  it  was  also  dependent  on  the  technologies 
from  other  programs  and  made  assumptions  about  the  availability  of  these  technologies  that 
were  not  realized. 

Summary  Comments 

AoAs  are  not  easy  to  conduct.  There  are  numerous  factors  to  consider  and 
numerous  organizations  to  please.  Factors  other  than  those  mentioned  above  that  must  be 
considered  that  complicate  the  analysis  include  the  following: 

•  alternatives  with  unequal  service  lives, 

•  systems  of  systems,  and 

•  alternatives  with  greatly  different  expenditure  profiles. 

In  addition  to  the  lessons  described  above,  the  AoA  must  realize  there  are  numerous 
organizations  in  the  approval  chain,  each  with  its  own  ideas  and  preferences.  It  is  important 
to  involve  these  organizations  early  in  the  AoA.  It  also  is  important  to  be  prepared  for  the 
changes  and  questions  that  will  inevitably  arise. 
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GAO  Has  Criticized  Past  AoAs 


*  Consider  a  narrow  range  of  alternatives 

*  Do  not  adequately  consider  cost,  schedule,  and 
performance  trade-offs 

*  Have  not  fully  understood  requirements 

*  Base  schedule  and  cost  estimates  on  optimistic 
assumptions 

*  Lack  sufficient  knowledge  of  technologies,  design 
risks,  and  production  impacts 

*  Do  not  have  adequate  guidance  from  OSD  and  the 
Services 
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Source:  Many  Analyses  of  Alternatives  Have  Not  Provided  a  Robust 
Assessment  of  Weapon  System  Options,  GAO-09-665,  September  2009 
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An  AoA  Requires  Answering  Three  Questions 
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How  Well? 
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Answering  These  Questions  Requires  Six 

Tasks 
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Important  Considerations  for  a  Successful  AoA 


*  Develop  a  thorough  study  plan 

*  Consider  a  range  of  alternatives  and  baselines 

*  Form  effective  relationships  with  oversight 
committees 

*  Conduct  trade-off  analyses  and  examine  sensitivities 

*  Have  a  flexible  analysis  methodology 

*  Display  results  that  are  easily  and  quickly 
understood 

*  Recognize  and  estimate  technical,  design,  and 
production  risks 
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A  Well  Thought  Out  Study  Plan  is  Key  to  a 

Successful  AoA 

*  Provides  guidance  and  objectives  for  conduct  of 
the  AoA 

*  Identifies: 

*  Scope  and  ground  rules 

*  Scenarios  and  CONOPS 

*  Draft  MOEs  and  MOPs 

*  Broad  classes  of  alternatives 

*  Range  of  costs  to  include 

*  Oversight  responsibilities 

*  Schedules 

*  Enables  a  consensus  among  all  stakeholders 
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An  Effective  Steering  Group  Will  Be  Necessary 
for  the  Successful  Conduct  of  the  AoA 

*  Role 

*  Provide  on-going  guidance  as  issues  arise 

*  Provide  access  to  stakeholders  and  experts 

*  Approve  study  director  decisions  as  work  progresses 
(e.g.  final  set  of  alternatives) 

*  Representation  (need  authority  to  make  decisions  or  can 
quickly  resolve  -  probably  equivalent  of  06  level) 

*  R&D  Centers 

*  Program  Office 

*  Cost  organization 

*  Operations 

*  Support  &  Logistics 

*  Technical  experts 

*  Administration 

*  Monthly  meetings  (present  issues,  discuss  progress,  get 
feedback) 

*  Government  needs  to  appoint  a  chairperson(s) 
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Trade-Off  Analyses  and  Sensitivities  Are 
Essential  for  Making  Informed  Decisions 

*  How  do  operational  requirements  drive  system 
performance? 

*  What  is  the  operational  impact  of  relaxing  various 
system  requirements? 

*  What  new  technologies  (and  associated  risks)  are 
needed  to  achieve  operational  goals? 

*  How  does  cost  relate  to  desired  system 
performance? 

*  Which  attributes  drive  costs  and  risks? 


RAND 


11  5/18/2012 


Choice  of  Propulsion  System  Impacts  Number 

of  Ships 
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Acquisition  Cost  (M$  FY08) 


Different  Force  Levels  Impact  Acquisition 

Costs 
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Number  of  Missiles 


Number  of  Missiles  Impact  Missile  Tube 

Diameter 
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Analysis  Methodologies  Must  Be  Flexible 


•  AoAs  must  recognize  things  will  change 

*  New  alternatives  and  scenarios  will  be  proposed 

*  New  technologies  will  be  available 

*  New  analysis  questions  will  arise 

*  Important,  new  MOEs  and  MOPs  will  emerge 

*  Analytical  methodologies  must  be  able  to  adapt  to 
those  changes 

*  Analysis  transparency  is  important 

*  Several  simple  models  are  often  better  than  a 
single,  complex  model 
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Analysis  Results  Must  Be  Displayed  In  a  Way 

That  is  Easily  Understood 

•  AoAs  typically  deal  with  complex  issues 

*  Interaction  of  several  variables 

*  Relationships  between  requirements, 
performance,  and  cost 

*  Displays  must  capture  the  complexity  but  clearly 
show  the  interdependencies 

*  Typically  best  to  keep  charts  simple 

*  Use  a  series  of  charts  to  build  to  the  primary 
points 
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Example  for  Multiple  Mission  Display 


Counter  Drug  Migrant  Coastal 
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Mission  Scorecard  for  MOPs  and  MOEs 


Alternative 

Measures  of  Effectiveness 

Measures  of  Performance 

%of  TOI 
Prosecuted 

%  of  Traffic 
Detected 

Small 

Boats 

Advanced  C4I 

Aviation 

A 

20% 

31% 

3 

Yes 

HH60/VUAV 

B 

15% 

27% 

2 

No* 

HH65 

C 

14% 

24% 

? 

No 

HH60/VUAV 

D 

17% 

27% 

? 

No 

HH60/VUAV 

E 

20% 

31% 

2 

Yes 

HH60/VUAV 

F 

15% 

26% 

2 

No 

HH65 

G 

20% 

31% 

2 

SWAP 

HH60/VUAV 

H 

18% 

29% 

2 

No 

HH60 

1 

18% 

29% 

2 

No 

HH60 

J 

18% 

29% 

2 

No 

HH60 

K 

18% 

29% 

3 

No 

HH60 

L 

18% 

29% 

2 

No 

HH60 
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Recognizing  and  Estimating  Risks  Are  an 

Important  Part  of  Ao As 

•  Ignoring  or  underestimating  risks  can  lead  to 
incorrect  decisions  and  future  cost  and  schedule 
growth 

*  Must  clearly  know  the  state  of  technologies 

*  Subject  matter/technical  experts  should  be  a 
part  of  the  AoA  team 

*  Risks  may  also  arise  from  the  industrial  base 
especially  the  second  tier  vendors 

*  Risks  may  also  arise  from  other  programs  the  AoA 
system  is  dependent  upon 
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Summary  Comments 


*  Keep  in  mind  other  complicating  factors 

*  Alternatives  with  unequal  service  lives 

*  System-of-systems 

*  Alternatives  with  greatly  different  expenditure 
profiles 

*  There  are  numerous  players  in  the  approval  chain 

*  They  will  have  their  own,  unique  ideas 

*  Involve  key  organizations  early 

*  Be  prepared  for  late  changes  and  questions 
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